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VISUALISING IMAGES 



This invention relates to methods and devices enabling a person to visualise 

images. 

The prospect of enabling subjects to visualise images through some external 
means, circumventing the human visual system, is one of clear significance. In 
particular, such a system might enable blind persons to "see", or at least assimilate some 
amount of visual information. Although systems based on echo location and on touch 
are known, there is at present no available system permitting optical images to be 
analysed in detail. 

The system described here translates visual images into sound. However 
unlike any of its predecessors such as US Patent US 5 097 326 it incorporates a system 
for feature extraction that is designed to enable a blind subject to selectively deconstruct 
a complex optical image into a set of simpler representations that make it easier for the 
subject to analyse. Each representation selectively isolates one or more features in the 
visual display which is then translated into an equivalent sound pattern. The blind 
subject listens and analyses these representations one at a time or in combination. and : 
uses the combination of sub- melodies to mentally reconstruct a representation of the 
visual target. Provision is also made to enable the subject to examine different portions 
of the image in a manner that bears some resemblance to the normal scan-path behaviour 
of the human eye, as it pursues a search pattern of foveations to explore a complex visual 
object (such as a face), selecting and gathering information in patch- like manner about 
salient features, which are then assembled to contribute collectively to the final percept 
and identification of the object. 
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US Patent US 5 097 326 discloses a system in which a visual image is 
pixellated and sinusoidal audio outputs are produced depending on the nature of each 
pixel. In this system, an audio output of characteristic frequency is associated with each 
pixel in a column of pixels, and the frequency of the audio output increases as one moves 
up a column of pixels. Furthermore, the amplitudes of each audio output from a column 
of Pixels are summated, and the sum is played to a listener. To the listener, this summed 
output can appear rather discordant. Furthermore, with complex images the output is 
extremely complex, making the detailed analysis of images very difficult. 

It is known that there are areas of similarity in the way the human brain 
interprets visual and auditory information. As with vision, auditory information is 
partitioned into discrete packages and conveyed to the relevant brain areas for separate 
processing : human speech sounds such as words and phrases are processed by 
Wernicke's area in the left hemisphere, where music is processed in the temporal lobe 
of the right hemisphere. In some animals, there are areas of the brain concerned with 
spatial perception in which both the visual and auditory topographical maps (involved 
in the location of objects in the environment by vision or by hearing) are superimposed. 

The approach to develop a sensory aid based on sensory substitution was 
guided by knowledge of the structure ands functional organization of the visual system 
and by neurophysiological, psychophysical and behavioral studies on visual pattern 
recognition in a variety of animals. For example, honeybees use a photographic memory 
to store and recall patterns in a (pixel by pixel fashion). However, they also distinguish 
patterns on the basis of global properties such as radial, circular, and bilateral symmetry. 
The honeybees visual system possesses spatial filters for detecting various symmetries. 
Radial and circular filters have also been discovered in primate vision. This suggests that 
information about the presence or absence of different types of symmetry in a visual 
target may sometimes be useful in the differentiation and characterization of visual 
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forms. In addition many insects that lack stereo vision use image-motion cues 
(differential movement parallax, changing size etc) to obtain information on depth. 
Image motion may also provide a powerful cue for segregating objects from their 
background, detecting imminent collisions, and for computing one's own motion. 
Likewise, the velocity flow field generated on the retina by different portions of a 3-D 
object as it approaches may produce sufficient information with which to reconstruct its 
form. These and other strategies such as visual velocity feedback for gaze and course 
control and the use of specialized scanning patterns for extracting information on specific 
visual features appear to be ubiquituous in vertebrate and invertebrate species where 
vision plays an important role in directing the animal's behaviour. Consequently any 
system of sense substitution that aims to compensate for the loss of vision, may well 
require to implement a set of strategies that are functionally similar to those employed 
by the visual system in its analysis of the spatial and spatiotemporal features of the 
subjects visual environment. 

The present invention is based on the surprising discovery that musical 
forms can be used to convey precise visual information to a subject. Such "precise" 
information can comprise spatial information such as the precise shapes of objects or 
symbols, and should be delineated from imagery which can be evoked in subjects 
listening to a favourite piece of music, in which instance images "brought to mind" by 
the music are personal in nature, and can vary quite dramatically from subject to subject. 
Almost certainly, this surprising discovery is related to the fundamental mechanisms 
governing the way in which the human brain segments, organises and processes 
information from various sources in multi dimensional space. However, discussion of 
such mechanisms is not the purpose of the present application. 

According to a first aspect of the invention there is provided a method 
enabling a person to visualise images comprising the steps of : 
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encoding spatial information relating to a feature or features contained 
within an image into the form of one or more musical sequences; and 

playing the musical sequence or sequences to the person. 

Subjects appear to be particularly receptive to the use of musical notes, a 
phenomenon which, as discussed above, may be due to the fundamental mechanisms of 
information processing in the human brain. 

"Spatial information" includes the shape, size, orientation and relative 
positions of features, as well as finer details such as surface decoration or, for example, 
the appearance of a face. As will be explained in more detail below, further visual 
information, such as colour and brightness, and temporal information, i.e. the movement 
of features, may also be visualised using the present invention. Features can be, for 
example, three dimensional objects, two dimensional objects such as drawings, or 
symbols such as letters, words and numbers. 

Features may be encoded by selecting a note or chord dependent on the 
distribution of the feature or features along an axis. 

The image may comprise a letter or a number, in which instance the method 
may be used to convey written information. 

The image may comprise the person's environment, in which instance the 
method may be used as a way of replicating some or all of the scene that the person 
would view if they were sighted. 



Spatial information may be encoded by: 
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representing the image as a two dimensional (2D) image; and 

forming one or more musical sequences, each comprising a series of notes 
or chords, in which i) each note or chord is selected dependant upon the distribution of 
the feature or features along a portion of the 2D image and ii) different notes or chords 
in a sequence correspond to different portions of the 2D image. 

The 2D image, or a portion of the 2D image, may be divided into a matrix 
of pixels, and i) each note or chord may be selected dependent upon the distribution of 
the feature or features along a column (or row) of pixels and ii) different notes or chords 
in a sequence may correspond to the distribution of the feature or feature along different 
columns (or rows) of pixels. A different note may be associated with each pixel along 
a column and, if a feature is recognised as being present in a pixel, the note 
corresponding to that pixel comprises part of the musical sequence. 

The method may enable a person to visualise moving features, and may 
comprise the step of playing a plurality of musical sequences corresponding to different 
positions and/or orientations of the moving feature. 

A subset of the full image may be encoded into the musical sequence or 
sequences. Predetermined features may be extracted from the image, and said 
predetermined features may be encoded into a musical sequence or sequences. In this 
way, and in contrast to the method of US 5 097 326, the information content of the image 
is reduced, and important features are highlighted. This renders the musical sequence 
or sequences played to the person much more tractable. 

A feature may be simplified by encoding a portion of the feature as a 
musical sequence. The feature may be encoded by encoding different portions of the 
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feature as different musical sequences. This approach greatly assists in the recognition 
of complex shapes; such shapes can be "broken down" into simpler sub-units. For 
example, one deconstructed representation might select only vertical lines present in the 
feature, and represent them in musical form. Another musical sequence might consist 
only of horizontal lines present in the feature. 

The image may be encoded into the form of a plurality of musical sequences 
which are played to the person as a melody. 

The image may be encoded as a plurality of musical sequences, each 
corresponding to different spatial resolutions. The image may be divided into two or 
more concentric zones, the zone at the centre of the image being encoded at the highest 
spatial resolution and the zone furthest from the centre of the image being encoded at the 
lowest spatial resolution. A feature or features may be visualised by obtaining a plurality 
of images in a sequence of saccadic movements. An example is the visualisation of a 
face, in which features such as eyes, nose and mouth are "scanned" at high resolution in 
a saccadic movement which mimics the operation of the human retina. 

The spatial resolution corresponding to a musical sequence may be 
indicated by the duration of the notes and/or chords in the sequence. 

The colour of the feature or features may be encoded by producing a 
musical sequence or sequences which comprise a plurality of different waveforms mixed 
in variable ratios, the waveforms being selected so that none of the waveforms may be 
created by a linear combination of the other two waveforms. Three waveforms may be 
mixed in variable ratios. The three waveforms, may be produced by filtering a master 
waveform between different frequency ranges. 
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The brightness of the feature or features may be encoded by varying the 
intensity of the musical sequence or sequences. 

According to a second aspect of the invention there is provided a device 
enabling a person to visualise images comprising: 



encoding means for encoding spatial information relating to the feature or 
features according to the first aspect of the invention; and 



playing means for playing the musical sequence or sequences to the person. 

The imaging means may comprise at least one video camera. 

The imaging means may comprise at least one charge coupled detector. 



The encoding means may comprise a microprocessor. 



The playing means may comprise an ear piece. 



The device may be portable, in which instance the imaging means may be 

hand-held. 

Methods and devices in accordance with the invention will now be 
described with reference to the accompanying drawings, in which:- 



imaging means for obtaining images of a feature or features; 



Figure 1 



shows a first 2D feature; 
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Figure 2 shows a second 2D feature; 

Figure 3 shows a sequence of locomotory motions of a bird; 

Figure 4 shows the division of an image field into a number of zones 
of different resolution; 



Figure 5 shows the coupling of different musical sequences in a 
melody; 

Figure 6 shows a) a solid object and b)processsed to highlight the 
contour of the solid object by image erosion; and 

Figure 7 shows a) a Necker cube, (right) a Necker cube with vertical 
lines removed, (b) sketches made by subject P.H., and c) 
sketches made by subject BK.; 



Figure 8 is a schematic diagram of an apparatus to enable a person to 
visualise images; 

Figure 9 shows recognition scores from subjects presented with 
complex geometrical shapes; 



Figure 10 illustrates segmentation of a complex object ( a house) to 
produce basic icons that a subject can recognise as a 
sequence of polyphonic melodies; and 



Figure 1 1 



shows the results from subjects presented with segmented 
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complex shapes. 

The invention comprises a method enabling a person to visualise images 
comprising the steps of: 

encoding spatial information relating to a feature or features contained 
within an image into the form of one or more musical sequences; and 

playing the musical sequence or sequences to the person. 

Figure 1 illustrates how the encoding may be performed on a computer 
generated 2D feature 10. The image 10 is, essentially, an arrangement of filled pixels 
which have been selected from a matrix of pixels. The musical sequence is produced by 
associating a different note with each pixel along a column of pixels and, if a feature is 
recognised as being present in a pixel, the note corresponding to that pixel comprises part 
of the musical sequence. If a feature occupies more than one pixel in a given column 
then a number of notes are played simultaneously, producing a chord. The entire image 
is encoded by performing this procedure for each column of pixels, thereby producing 
a sequence of notes or chords. 

In Figure 1, this encoding procedure is performed using a moveable cursor 
12. The cursor 12 is divided into 32 segments corresponding to the notes in four octaves 
of the scale of C major (which comprises, in ascending order, the eight notes C D E F A 
B C). The cursor 12 defines a Y axis. Thus, features are encoded by selecting a note or 
chord dependent on the distribution of the feature along the Y axis. Encoding proceeds 
by moving the cursor 12 along the X axis, from left to right as viewed in Figure 1 . Each 
movement of the cursor 12 samples a new column of pixels. If the cursor 12 encounters 
one or more filled pixels (corresponding to a portion of the feature 10) then the note or 
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notes corresponding to the segments of the cursor 12 which have encountered the filled 
pixels are played. Thus, if the cursor 12 is moving across the screen the screen at a 
velocity of p columns of pixels per second, the time between the playing of successive 
notes or chords is 1/p seconds. 

In other words, spatial information corresponding to the shape of the figure 
in Cartesian coordinates is encoded, or transposed, into a musical sequence in which the 
Y ordinate is represented by musical notes and the X ordinate by time. 

Returning to the specific example shown in Figure 1, it can be seen that 
movement of the cursor 12 over the figure 10 will result in the playing of a musical 
sequence in which the ascending notes G, A, B, C of the second lowest octave and C, D, 
E, F of the second highest octave are played in succession. 

As a further example, Figure 2 shows a computer generator square figure 
20 which also comprises a number of filled pixels. When the cursor 12 is moved across 
the square 20, the first component of the musical sequence is a chord which comprises 
the ten notes A, B, C of the second lowest octave and C, D, E, F ? G, A, B of the second 
highest octave. The number of notes involved results in a chord which gives the 
impression of density or thickness. The next eight components of the musical sequence 
are chords in which only two notes, A and B, are played, these notes corresponding to 
the top and bottom sides of the square 20. The result is a sound which might be 
described as "thinner". The final component of the musical sequence is the chord 
comprising the ten notes. 

The use of the scale of C major (which does not contain flats or sharps) is 
not limiting: other musical scales may be used. Indeed, since the four octaves utilised 
in the above examples represents, approximately, the range of human hearing, thus 
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limiting the Y axis resolution of the encoded image, it may be advantageous to utilise the 
chromatic scale. In principle, the image might be encoded using a different coordinate 
system than Cartesian coordinates, such as polar coordinates. 

A computer program was written in the C++ language, running under the 
Windows (RTM) operating system to enable 2D shapes and objects to be encoded using 
the approach described above. A Musical Instrument Digital Interface (MIDI) allowed 
interfacing to a sound card in order to play the musical sequence. Confidential tests were 
performed, using the software, on a number of blind subjects, and on (blind folded) 
sighted subjects. Extremely favourable results were obtained in tests which employed 
a variety of geometric shapes and letters. For example, subjects were quickly able to 
read simple words, having been trained on individual letters. Furthermore, subjects were 
able to recognise figures consisting of one geometric shape contained within another 
shape, (such as a triangle within a square), having been trained on the individual 
component geometric shapes. Figure 9 shows the recognition scores obtained to a series 
of geometric shapes of differing complexities. 

Global perception of animal perception translated into species specifi c rhythmical 
pattern of sound 

In a further development, it is possible to visualise moving features by 
playing a plurality of musical sequences corresponding to different positions and/or 
orientations of the moving feature. In this way, dynamical information can be visualised 
in a way which bears similarities to the principles of cinematography, in which 
successive frames showing different stages of the movement are shown. 

The computer program described above was adapted to produce a series of 
images which simulate the locomotory sequence of limb movements displayed by a 
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variety of animal species, namely i) a galloping horse; ii) a running cheetah, iii) a 
walking man; iv) a flying bird; v) a swimming fish; vi) a bipedal lizard; vii) a 
quadrupedal lizard; viii) a wriggling worm; and ix) a crawling locust. Figure 3 shows 
a sequence of images containing the feature of a flying bird. 

Successive musical sequences, corresponding to different "frames" in 
sequences of images such as that shown in Figure 3, were played to subjects in the 
confidential tests. Blindfolded normal and previously sighted blind subjects were able 
to distinguish between the different locomotory motions and, in some instances, were 
able to correctly identify a locomotory motion with no previous training using the 
locomotory motion. 

Cross-modality perception of complex stationary visual images translated into 
sound 

We now turn to the problem of encoding more complex images. The 
approach adopted is to substantially mimic, in a number of aspects, the operation of the 
human eye. 

An important aspect of the present invention is the realisation that it is often 
advantageous to encode a subset of the full image into a musical sequence or sequences. 
Predetermined features can be extracted form the image, and said predetermined features 
may be encoded into a musical sequence or sequences. Feature extraction or pattern 
recognition algorithms can be used for this purpose. 

For complex images it was necessary to extend the battery of search 
strategies available to our subjects to enable them to carry out a satisfactory exploration 
and analysis of the target. These included (1) a facility to mask or block out areas areas 
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of the image to enable the subject to carry out a patch by patch exploration and analysis 
of the target and to selectively examine portions of the image. This was intoduced in an 
attempt to simulate the tactile search pattern that a blind subject normally carries out 
when tactaully exploring the shape of a solid object. (2) The second major change was 
to carry out of a strategy of feature extraction to segment and produce a set of simplified 
representations of the image. In our original scheme, the entire image of the "object" 
was displayed on the computer screen and all pixels constituting the image were thereby 
primed to activate the release of the appropriate musical notes (that signalled their 
respective locations on the screen) when they were intercepted by the moving tracker bar. 
To be fully primed it was necessary and sufficient, only that a pixel be occupied by part 
of the image on the screen. To facilitate the analysis of more complex visual shapes we 
introduced several additional conditions that needed to be fulfilled before a pixel is fully 
primed to activate the tracker bar. Effectively this is equivalent to having the full image 
of the "object" appearing on the monitor screen simultaneously by several different 
spatially distributed sets of feature detectors that each select only those pixels included 
in those parts of the image that display the selected feature to which a particular array of 
detectors is tuned. Thus one array of orientation selective detectors may select for 
priming any set of pixels that lie on well defined vertical lines in the image, while 
another array of feature detectors may do the same for horizontal lines etc. Further 
provision is made to allow the subject the choice (i) of selecting which feature (o r 
combination of features) of the image are to be presented and (ii) which "voices" are to 
be excluded during a particular presentation. By screening the image with several sets 
of detectors which select for different features (e.g. horizontal, vertical and oblique line 
components), we can generate musical patterns associated with an image in which all but 
one of the features have been removed or an image that combines simultaneously, a 
number of selected features (such as vertical and horizontal line components, or for 
sounding the whole figure i.e. by priming all the pixels and by arranging to scan and 
sound all the "voices" (segmented parts of the melody) simultaneously. Effectively this 
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allows the subject to isolate and separately examine the contents of several segmented 
packages in which different features of the image have been sequestered. 

Feature Extraction and Image Simplification 

In the case of complex images, it is often confusing to a listener if the full 
image is encoded, because the associated musical sequence is too complex. By encoding 
only certain features, it becomes easier for a subject to distinguish, or resolve, 
identifiable features. There are a numbers of ways in which the encoded information 
might be presented to a listener. One way is to play a single, simplified musical 
sequence to the person. Alternatively, numerous musical sequences might be played: 
there are a number of ways in which a plurality of musical sequences can be presented 
to a person, and these are discussed below. 

Individual features present in an image can themselves be of complicated 
shape and form. A feature can be simplified by encoding a portion of the feature as a 
musical sequence. 

Eroding the Image to Sharpen and Extract Object Contents 

One example of this is if a feature is a solid object, such as the object 60 
shown in Figure 6a. Encoding the entire object 60 into a musical sequence would result 
in a long series of chords each containing multiple notes. It is difficult for a subject to 
distinguish finer details of the scope of the object 60 from such a musical sequence. It 
is advantageous to encode only the edges of the object 60 as a musical sequence, 
resulting in a sequence which is more tractable to the subject. This can be achieved by 
pre-processing the object 60 to produce an outline object 62 (Figure 6b). The outline 
object 62 is then encoded. 
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Another example of feature simplification is provided by encoding different 
portions of a feature as different musical sequences. The different portions can comprise, 
for example, vertical lines, horizontal lines or sloping lines present in the feature. 

Example 

Figure 7 shows a Necker cube 70, which is a two dimensional 
representation of a wire model of a cube. In confidential tests, subjects were presented 
with the musical sequence associated with the Necker cube. Before the presentation each 
subject told that the feature was a 2-D representation of a 3-D wire model of some object, 
and they were required to analyse the sound pattern, and from this, to reconstruct its 
form. A sighted subject PH was given a drawing pad and invited to sketch the shapes 
he inferred from listening to the sound pattern during the process of reconstruction (see 
Figure 7b). A blind subject BK was reluctant on this occasion to attempt any drawing 
while performing the task but did produce one later when asked how he had analysed the 
shape. 

Test procedure for PH. (i) Subject was allowed to listen to a series of 
presentations of the whole Necker cube figure 70. He reported that the task was 
"difficult", (ii) Experimenter suggests he simplify the figure and did so by removing the 
vertical struts and presenting the subject with the two rhomboid surfaces 72, 74 depicted 
in Figure 7a (right). After listening to a number of presentations of the deconstructed 
figure the subject asked for the original whole figure to be played back to him. PH then 
produced a series of sketches in which he reconstructed the Necker cube. 

Test procedure for BK. The subject was given the same instructions as PH 
except that on this occasion, BK chose not to draw but to analyse and reconstruct it 
mentally and give a verbal description. After listening to several (sound) presentations 
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of the entire Necker cube figure, and without it being segmented or disassembled, BK 
declared that he was able to analyse the basic line elements and their spatial arrangement 
relative to one another, whereupon he identified the figure as a "cube". When asked later 
to illustrate how he had done this he produced the set of drawings depicted in Figure 7c. 

The amount of feature simplification can be varied as a user learns to 
recognise more complicated structures. Thus, once a user has learnt to recognise these, 
they can be used by the subject as building blocks with which to analyse more complex 
figures. A user might learn to recognise a feature by listening to a portion of the feature, 
and then returning to the musical sequence corresponding to the entire feature, such as 
described in the Example. Alternatively, the entire feature might be encoded by 
encoding different portions of the feature as different musical sequences. In a trivial 
example, a square might be encoded by forming one musical sequence corresponding to 
the two vertical lines of the square, and forming another musical sequence corresponding 
to the two horizontal lines. The two musical sequences might be played to a subject in 
series, i.e., sequentially, or, more subtly, in parallel. 

Representing and processing images to different levels of resolution 

Figure 4 shows how an image might be divided into a number of concentric 
domains or zones 40, 42, 44, 46, for the purpose of encoding an image. The spatial 
resolution, i.e. the size of the pixels used to produce a musical sequence, is different in 
each zone 40, 42, 44, 46. More specifically, the image defined by the largest zone 40 is 
encoded at the lowest resolution, zone 42 corresponds to a medium resolution, zone 44 
corresponds to a medium/high resolution, whilst the image defined by the smallest zone 
46 is encoded at the highest resolution. The use of four zones in this way is similar to 
the structural divisions of the fundus of the human eye, in which visual activity is highest 
at the foveola, in the centre of the retina, and diminishes going radially from the centre 
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of the retina, through the foveola, fovea, parafovea and perifovea. The use of four 
concentric zones is not limiting : different numbers of zones might be employed. In 
principle, the zones need not be concentric, although, for reasons outlined below, this 
configuration is strongly preferred. 

This constitutes preprocessing which utilizes a selective process of feature 
extracting modules to first deconstruct the image into separate packages; then convert 
them into equivalent musical phrases and presents these to the subject as the several 
"voices of a polyphonic melody", and is an extremely important tool for analysing the 
structure of complex images. 

Basic icons extracted can be used in the analysis of simple shapes 

The visual system of mammals and primates, contains neurons sensitive to 
line orientation that are an important subset of the feature selective elements in the brain 
that play a key role in the analysis and perceptual representation of visual forms. In the 
immature brain they and other feature detectors, may represent the functional building 
blocks (basic icons) for the later assembly, through learning, of more elaborate and 
sophisticated feature detectors (learned icons), that preprocess visual information in 
parallel and are used by the experienced adult brain, to speed up analysis and perception 
of the visual world. Following this line of reasoning we supplied our subjects (blind, 
and normal (blindfolded) alike) with a verbal description of the musical equivalent of a 
set of basic line elements which they were instructed to utilise to analyse the 
characteristic sound signatures generated by different shapes and to use this information 
to reconstruct the visual form. They were also encouraged to explore the figure for 
symmetrical features that may help them in the task of visual contour identification. 

It will be apparent that it is possible to observe a scene at low resolution, 
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and then to pan it so that individual features can be resolved at higher resolution 
receptive fields. Furthermore, the configuration of Figure 4 might be used to view a 
scene or object in saccadic fashion. Saccadic eye movements are rapid, ballistic 
movements of the eyes used in scanning a scene or object. They are used to locate the 
portion of the image of interest onto the fovea i.e. the zone of the retina with the finest 
granularity and resolution. An example is facial recognition in which the eye motion 
rapidly and successively puts features such as eyes, nose and mouth in the central, high 
resolution foveola and fovea zones of the retina. Due to time taken in playing a musical 
sequence, such saccadic movements using the present invention will be less rapid than 
in the human eye. However, it is quite feasible that a low resolution image, encoded 
using the present invention, might be used to indicate features of interest which are 
successively brought into the central portion of the image for visualisation at high 
resolution. 

From the foregoing, it will be apparent that image or images can be 
segregated into a plurality of musical sequences, corresponding, for example, views at 
different resolutions, saccadic compilations of several related images, and "special" 
sequences relating to certain "programmed" features. These separate auditory 
representations may be bound into a single percept by the use of melody. Consider an 
example in which the outline of a house is presented on one sequence, followed by tht 
finer details of features (Figure 1 0). 

The image of the house would then be encoded in a (repeated) musical 
melody in which the outline of the house featured as the contents of one window, and 
other features etc contained in the other. 

To generate the "melody" (which encompasses the contents of both 
windows) and to preserve a sense of the "spatial" relationships between them we 
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introduce two additional modifications: 



(i) 



Figure 5a illustrates how to alternate the presentations by 



modulating the sound intensity of each window presentation 50 ? 52 above and below the 
auditory threshold and arranging for the intensity modulation of the two presentations 50, 
52 to be 1 80 degrees out of phase. 



(ii) Figure 5b shows the continued modulation of the sound intensity of 



presentation 50, symmetrically about the auditory threshold. For the other presentation 
52 we shift the baseline of its modulation a fraction up, so that it lies some distance 
above the auditory threshold. Thus we have introduced: 



(a) a tempo which musically brackets the contents of the two windows 
within the simple melody 

(b) a short interval at the start and end of each cycle when a portion of 
each window may be heard simultaneously - one rising in intensity, 
the other decreasing. 



Thus by simply shifting the modulation baseline of one or the other above 
or below the auditory threshold one may selectively present and listen to: 



(i) the contents of one of the "voices". 

(ii) generate a full alternation of the contents of each window. 

(iii) a tempo that brackets the contents of the two windows: allowing the 
observer to perceive them as a single entity (much as the theme tune 
of a song). 
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(iv) portions of both windows simultaneously. 

Figure 1 1 illustrates the responses to such an approach by subjects tested 
on segmented complex images (a boat, a staircase, a house, a castle). 

The confidential experiments demonstrate that blind (previously sighted but 
lacking light perception) and blindfolded(normal) subjects can utilise the sound 
representations of visual images to analyse the structure of novel (and relatively 
complex) visual forms. They achieve this by deconstructing the figure into its 
constituent line elements and from there proceed to a fairly accurate reconstruction and 
description of the whole figure. Whilst in some instances they failed to correctly name 
the object, they were nevertheless still able to produce an accurate description or sketch 
of the figure presented to them (e.g. the house and castle in Figure 1 1). This shows that 
we were not simply providing the subject with a dictionary of complex sounds that are 
associated with specific objects but are providing them with the tools with which to 
analyse novel complex objects in terms of simpler building blocks (e.g. basic line 
elements) whose associated sound signatures they had either learned previously or 
appeared to recognise naturally ( Figures 1,2 ). Once subjects have successfully carried 
out an analysis and identified a visual form we find they progress to being able to 
recognise and isolate more complex features of the object which they subsequently us^ 
as higher level cues (new feature detectors that are established through learning) to 
facilitate recognition on subsequent presentations of the object (and in some cases 
becoming a high level feature detector whose selectivity is for the object itself). 
Whatever the sensory modality, the use of feature detectors to segment a complex 
stimulus pattern, seems to be the key to analysing (and reconstructing) novel complex 
shapes. 

This approach can be extended using principles well established in 
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musicology in order to generate and modulate complex melodies with many subsidiary 
"voices". Each "voice" contains visual information, and furthermore, visual information 
can be contained in the relationships between the "voices". The individual musical 
sequences might utilise different waveforms, i.e. different instruments or different voices 
might be allocated to different musical sequences, giving rise to considerations of 
harmony. 

The colour of features can be represented using the present invention. In 
classical colour theory, the perception of colour is generated through the differential 
absorption of different wavebands of light by the visual pigments contained in 3 types 
of photoreceptor which serve as primaries. Any 3 coloured lights can serve as primaries 
provided only that when mixed together in suitable proportions they produced the 
sensation of "white" and perhaps more importantly: on condition that it should not be 
possible to match one of those by linear combination of the other two. 

One way of achieving such colour mixing with the present invention is to 
select a master waveform corresponding to a musical instrument which spans a 
reasonable range of octaves and whose notes contain a rich range of harmonic overtones. 
A triad of primary waveforms, for every notes in the span of octaves employed in the 
musical sequences, is generated as follows: 

(i) To generate the "long wavelength" version of that note we use a 
sound bandpass filter to remove some of the medium and higher frequency components 
from the sound normally generated by that note. 



(ii) For the "medium wavelength" version of the same note we follow 
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the same procedure except that we filter out some of the high and low frequency 
components. 

(iii) For the "short wavelength" version we filter out some of the low and 
medium frequency components. 

(iv) Next we adjust the relative intensities of the triad generated so that 
when they are sounded together they retrieve the sound generated by the original note 
(which serves as the achromatic note in the set). 

(v) This triad should satisfy the principal condition required of a set of 
colour primaries: namely that it should not be possible for the sound of any one of the 
triad to be matched by a suitable (intensity) mixture of the other two. 

The procedure described above is repeated for each note in the entire set of 
octaves utilised by the present invention. However to obtain a suitable set of triads for 
each note, the characteristics of the bandpass filters need to be altered in each case to take 
account of the change in pitch of the zero harmonic (fundamental) as we progress up or 
down the scale of notes. 

An alternative method - the method of choice - is to employ three different 
musical instruments that differ in the sounds produced but having the same fundamental 
frequency encoding pixel position. Colour coding may be achieved by mixing the three 
harmonic sets in desired proportions. Thus a point can be represented both by the 
position and by colour together. 



Spatial characteristics of visual displays 
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The system may also be used to translate image motion cues (such as 
differential movement, parallax, changing size) into equivalent sound patterns which can 
be utilised to segregate figure from background in 3D and produce information on the 
relative depth between visual objects. 

The present invention provides devices enabling a person to visualise 
images. Such devices comprise: imaging means for obtaining images of a feature or 
features; encoding means for encoding spatial information relating to the feature or 
features in the manner described above; and playing means for playing the musical 
sequence or sequences to the person. 

The imaging means can comprise a video camera, although other means, 
such as CCD or photovoltaic detectors, might be employed. The encoding means 
performs the functions of analysing the image produced by the imaging means in a 
suitable manner, and encoding the analysed image into suitable musical sequences. The 
analysis step might comprise the division of the image, or portions of the image, into the 
desired number of pixels. It is of course highly desirable that the device is portable, and 
thus a small, dedicated microprocessor might be used as encoding means. A small video 
camera can be used as part of a portable device: the video camera can be incorporated 
into a hand-held "wand. In both instances, scanning movements can be accomplished 
by the person via hand motion or automatically. The playing means can comprise an ear- 
piece worn by the person. 

Figure 8 shows a system comprising a CCD camera 80 which captures 
images and relays the data to a personal computer (PC) 82. The PC 82 is adapted, 
through the use of suitable software, to perform certain preprocessing of the data and to 
generate, in real time, appropriate musical sequences using the MIDI protocol. The PC 
82 is interfaced to a sound card (not shown) which permits musical sequences to be 
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played on a loudspeaker 84. 

Preprocessing of the image captured by the camera 80 proceeds according 
to the following scheme: 

1. The image is positioned on a grid comprising a plurality of pixels. 
The pixel size is variable, thereby permitting variation of the spatial 
resolution. 

2. The image is thresholded, so that each pixel in the grid is either 
black or white. Selection of achromatic pixel colour (black or 
white) is dependent on whether intensity of the image within the 
pixel exceeds a predetermined threshold. 

3. The edges of the features present in the image are detected. Various 
filters for these purposes are know in the art: in this embodiment, a 
Sobel filter or a Laplace filter can be used. 

4. "Erosion" of solid objects is performed to produce an outline object 
of the type described previously in relation to Figure 6b. Th^ 
resulting outline object consists only of edges: this processing 
simplifies the musical sequence associated with the image, and 
improves subsequent shape or feature recognition steps. 

5. Musical notes are assigned in order to encode the image. This is 
achieved by using a net of relatively large "grid elements" in which 
each grid element contains a number of pixels. The average 
intensity of the pixels in a grid element is calculated, and a note is 
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assigned to the grid element only if a predetermined intensity 
threshold is exceeded. This process reduces the spatial resolution of 
the encoded images, but, very usefully, reduces the effect of noise 
in the image. In this non-limiting embodiment, a total of 53 white 
scale notes are employed, i.e., the Y axis of the encoded image 
comprises 53 sub-divisions. 

The software offers numerous additional facilities. Shape recognition 



algorithms can be selected to recognise, for example, edges, comers, circles, straight 
lines etc. Boolean logic operators AND and OR can be employed to compare 
consecutive images, either before or after preprocessing. This is useful inter alia for 
detecting movement of objects. 

Due to the digital nature of the data, it is possible to zoom in and out of the 
image using the software. 



It would be possible to produce a portable version of this system by 



replacing the PC 82 with a compact unit comprising a dedicated microprocessor. 
Musical sequences can be played to a user with an ear piece. 
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CLAIMS 

1 . A method enabling a person to visualise images comprising the steps of: 

encoding spatial information relating to a feature or features contained 
within an image into the form of one or more musical sequences; and, 

playing the musical sequence or sequences to the person. 

2. A method according to claim 1 in which spatial information is encoded by 
selecting a note or chord dependent on the distribution of the feature or features along 
an axis. 

3. A method according to claim 1 or claim 2 in which the image comprises a 
letter or a number. 

4. A method according to any of claims 1 to 3 in which the image comprises 
the person's environment. 

5. A method according to any of the previous claim in which spatial 
information is encoded by: 

represent the image as a 2D image 

forming one or more musical sequences, each comprising a series of notes 
or chords, in which i) each note or chord is selected dependent upon the 
distribution of the feature or features along a portion of the 2D image and 
ii) different notes or chords in a sequence correspond to different portions 



WO 99/58087 PCT/GB99/01506 

-27- 



of the 2D image. 

6. A method according to claim 5 in which the 2D image, or a portion of the 
2D image, is divided into a matrix of pixels, and i) each note or chord is selected 
dependent upon the distribution of the feature or features along a column (or rows) of 
pixels and ii) different notes or chords in a sequence correspond to the distribution of the 
feature or features along different columns (or rows) of pixels. 

7. A method according t o claim 6 in which a different note is associated with 
each pixel along a column and, if a feature recognised as being present in a pixel, the 
note corresponding to that pixel comprises part of the musical sequence. 

8. A method according to any of claims 1 to 7 enabling a person to visualise 
moving features comprising the step of playing a plurality of musical sequences 
corresponding to different positions and/or orientations of the moving feature. 

9. A method according to any previous claims in which a subset of the full 
image is encoded into the musical sequence or sequences. 

10. A method according to claim 9 in which predetermined features arc. 
extracted from the image, and said predetermined features are encoded into a musical 
sequence or sequences. 

11. A method according to any previous claims in which a feature is simplified 
by encoding a portion of the feature as a musical sequence. 
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12. A method according to claim 11 in which the feature is encoded by 
encoding different portions of the feature as different musical sequences. 

13. A method according to any of the previous claims in which the image is 
encoded into the form of a plurality of musical sequences which are played to the person 
as a melody. 

14. A method according to any of the previous claims in which the image is 
encoded as a plurality of musical sequences, each corresponding to different spatial 
resolutions. 

15. A method according to claim 14 in which the image is divided into two or 
more concentric zones, the zone at the centre of the image being optionally encoded at 
the highest spatial resolution and the zone furthest from the centre of the image being 
encoded at the lowest spatial resolution. 

16. A method according to claim 1 5 in which a feature or features are visualised 
by obtaining a plurality of images in a saccadic-like series of movements, functionally 
equivalent to a scan path that the eye follows when it examines an object. 

17. A method according to any of claims 1 4 to 1 6 in which the spatial resolution 
corresponding to a musical sequence is indicated by the duration of the notes and chords 
in the sequence. 

18. A method according to any of the previous claims in which the colour of the 
feature or features is encoded by producing a musical sequence or sequences which 
comprise a plurality of different sets of waveforms corresponding to the harmonics of 
different sound instruments mixed in variable ratios. 
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19. A method according to claim 18 in which three waveforms are mixed in 
variable ratios. 

20. A method according to claim 1 9 in which the three waveforms are produced 
by filtering a master waveform between different frequency ranges. 

21. A method according to any of the previous claims in which the brightness 
of the feature or features is encoded by varying the intensity of the musical sequence or 
sequences. 

22. A method according to any previous claims in which image motion cues are 
translated into equivalent sound patterns which are utilised to segregate foreground 
features from background in 3D and produce information on the relative depth between 
features. 

23. A device enabling a person to visualise images comprising: 

imaging means for obtaining images of a feature or features; 

encoding means for encoding spatial information relating to the feature er 
features according to any of claims 1 to 22; and 

playing means for playing the musical sequence or sequences to the person. 

24. A device according to claim 23 in which the imaging means comprises at 
least one video camera. 



25. 



A device according to claim 23 in which the imaging means comprises at 
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least one charge coupled detector. 

26. A device according to any of claims 23 to 25 in which the encoding means 
comprises a microprocessor. 

27. A device according to any of claims 23 to 26 in which the playing means 
comprises an ear-piece. 



28. A portable device according to any of claims 23 to 27 in which the imaging 

means is hand-held. 
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EPODOC. WPIL 



C. DOCUMFNTS CONSIDERED TO BE RELEVANT 



Category* 



Citation of document, with indication, where appropriate, of the relevant passages 



Relevant to claim No. 



WO 8200395 Al (THALES RESOURCES, INC.), 

4 February 1982 (04.02.82), the whole document 



BE 1003022 A3 (UNIVERSITE CATHOLIQUE DE LOUVAIN), 
29 October 1991 (29.10.91), the whole document 



EP 0410045 Al (N.V. PHILIPS' GLOEILAMPENFABRIEKEN) , 
30 January 1991 (30.01.91), figure 2, 
abstract 



1-13,21, 
23-25,27 

14,15,26,28 



14,15,26,28 



1-21,23-28 



yj Furl her documents arc listed in the continuation of Box C. | y] See patent family annex. 



♦ Special categories of cited document* 

"A" document defining ihe general slate of the art which is not considered 

to he of particular rci cvancc 
")*" crlicr document hul puhlishcd on or after the international filing date 

*!." document which may Uirow doubts on priority claim(s) or which is 
a ted in establish the publicauon date of another citauon or other 
special reason (as specified) 

"O" document referring to an oral disci osurc, use, exhibition or other 
means 

"P" document puhlishcd prior to the intern auonaJ filing date hut later than 
the priority date claimed 



T" later document published alter the international filing date or priority 
date and not in con ilia with the application but a ted to understand 
the principle or theory underlying the invention 

"X" document of particular relevance: the claimed invention cannot be 
considered novel or cannot be considered to involve an invenuve 
step when the document is taken alone 

" Y" document of particular relevance: the claimed invention cannot be 
considered to involve an inventive step when the document is 
combined with one or more other such documents, such combination 
being obvious to a per s on skilled in the art 

document member of the same patent family 



Hale uf the actual completion of the international search 

2 November 1999 


Date of mailing of the international search report 

07. 12. 1999 


Name ana mailing address erf the tmernauonaJ Searching Auihoray 

European Patent Office P. 8. 5818 Patoffllaan 2 

NL-2280 HV Rifswttk 

Te«*31 -70)340-2040. Tx 31 651 «ponl. 

Fart+3 1-70)340-30 16 


Authorized officer 

Oliver Piolat / MR 
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C (Continuation). DOCUMENTS CONSIDERED TO BE RELEVANT 




Category* 


Citation of document, with indication, where appropriate, ofthe relevant passages 


Relevant to claim No. 


A 


US 5310962 A (M. KIMPARA ET AL), 10 May 1994 

^ ID . Ub . y*r J , figure d, aostraCL 


1-21,23-28 


A 


US 4322744 A (A.N. STANTON), 30 March 1982 

/on f\ o 09S f initvio Q aKc+v»ar*1" 

^oU.Uo.Oii^, t 1 gure o, dDSLratt 


1-21,23-28 


A 


US 4000565 A (A.W. OVERBY ET AL), 4 January 1977 
(04.01.77), abstract 


1-21,23-28 


A 


FR 2596940 Al (P. DE TIEGE) , 9 October 1987 
(09.10.87), figure 1, abstract 


1-21,23-28 



I-orm PC;r'1SA/2I() (continuation of second sheet) (July 1992) 



o 

INTERNATIONAL SEARCH REPORT 



IntS^PionaJ application No. 

PCT/GB 99/01506 



Box I Observations where certain claims were found unsearchable (Continuation of item 1 of first sheet) 

This International Search Report has not been established in respect of certain claims under Article 17 (2) (a) for the following reasons: 
1. Claims Nos.: 

because they relate to subject matter not required to be searched by this Authority, namely: 



jlT] Claims Nos.: 22 

because they relate to parts of the International Application that do not comply with the prescribed requirements to such 
an extent that no meaningful International Search can be carried out, specifically: 

The matter claimed in claim 22 is not clear. See articles 5 and 6. 



3. Claims Nos.: 

because they are dependent claims and are not drafted in accordance with the second and third sentences of Rule 6.4(a). 

Box II Observations where unity of invention is lacking (Continuation of item 2 of first sheet) 

This International Searching Authority found multiple inventions in this international application, as fellows: 



1 . I I As all required additional search fees were timely paid by the applicant, this International Search Report covers all 
' ' <t ft arch able claims. 



2. As all searchable claims couid be searched without effort justifying an additional fee, this Authority did not invite payment 

of any additional fee. 



3. | I As only some of the required additional search fees were timely paid by the applicant, 
' ' covers only those claims for which fees were paid, specifically claims Nos.: 



this international Search Report 



4. I I No required additional search fees were timely paid by the applicant. Consequently, this International Search Report is 
restricted to the invention first mentioned in the claims; it is covered by claims Nos.: 



Remark on Protest j [ The additional search fees were accompanied by the applicant's protest. 

~ | [ No protest accompanied the payment of additional search fees. 
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INTERNATIONAL SEARCH REPORT 

liifoim.ition on patent family members 

28/09/99 


International application No. 

PCT/GB 99/01506 


Patent document 
cited in search report 


Publication 
dale 


Patent family 
membcr(s) 


Publication 
date 



WO 8200395 Al 



04/02/82 



AT 18841 T 

AU 546019 B 

AU 7376881 A 

CA 1165447 A 

EP 0055762 A,B 

US 4378569 A 



15/04/86 
08/08/85 
16/02/82 
10/04/84 
14/07/82 
29/03/83 



BE 1003022 A3 
EP 0410045 Al 



29/10/91 
30/01/91 



NONE 



US 5310962 A 



10/05/94 



Jp 


3184540 


A 


12/08/91 


us 


5097326 


A 


17/03/92 


JP 


63309499 


A 


16/12/88 


us 


5159140 


A 


27/10/92 


JP 


1070797 


A 


16/03/89 


JP 


2086674 


C 


02/09/96 


JP 


8003715 


B 


17/01/96 


JP 


1091189 


A 


10/04/89 


JP 


2629740 


B 


16/07/97 


JP 


1091190 


A 


10/04/89 


JP 


1091188 


A 


10/04/89 


JP 


2508136 


B 


19/06/96 


us 


4913297 


A 


03/04/90 



US 4322744 A 30/03/82 NONE 



US 


4000565 


A 


04/01/77 


NONE 


FR 


2596940 


Al 


09/10/87 


NONE 
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INTERNATIONAL PRELIMINARY EXAMINATION REPORT 

Tecnnolo 

(PCT Article 36 and Rule 70) 
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ogy Center c&oo 



Applicant's or agent's file reference 
M98/0209/PCT 



FOR FURTHER ACTION 



See Notification of Transmittal of International 
Preliminary Examination Report (Form PCT/IPEA/416) 



international application No. 
PCT/GB99/01506 



International filing date (day/month/year) 
12/05/1999 



Priority date (day/month/year) 
12/05/1998 



International Patent Classification (IPC) or national classification and IPC 
A61F9/08 



Applicant 

UNIVERSITY OF MANCHESTER INST...TECHNOLOGYetal. 



1 . This international preliminary examination report has been prepared by this International Preliminary Examining Authority 
and is transmitted to the applicant according to Article 36. 

2. This REPORT consists of a total of 8 sheets, including this cover sheet. 

Q This report is also accompanied by ANNEXES, i.e. sheets of the description, claims and/or drawings which have 
been amended and are the basis for this report and/or sheets containing rectifications made before this Authority 
(see Rule 70.16 and Section 607 of the Administrative instructions under the PCT). 

These annexes consist of a total of isheets. 



3. This report contains indications relating to the following items: 

I S Basis of the report 
Priority 

Non-establishment of opinion with regard to novelty, inventive step and industrial applicability 
Lack of unity of invention 

Reasoned statement under Article 35(2) with regard to novelty, inventive step or industrial applicability; 
citations and explanations suporting such statement 

Certain documents cited 

Certain defects in the international application 



II 


□ 
IS 


III 

IV 


□ 


V 




VI 


□ 


VII 




VIII 


□ 



Date of submission of the demand 
08/12/1999 



Date of completion of this report 
08.09.2000 



Name and mailing address of the international 
preliminary examining authority: 

- European Patent Office 
D-80298 Munich 

Tel. +49 89 2399---d^Tx: 523656 epmu d 

^--Fax: +49*89 2399 - 4465 



Authorized officer 
Krantz, L 

Telephone No. +49 89 2399 2523 




Form PCT/I PEA/409 (cover sheet) (January 1994) 



V 



LE22731. 05.09.2000 



INTERNATIONAL PRELIMINARY 
EXAMINATION REPORT 



International application No. PCT/GB99/01 506 



1. Basis of the report 

1 . This report has been drawn on the basis of (substitute sheets which have been furnished to the receiving Office in 
response to an invitation under Article 14 are referred to in this report as "originally filed" and are not annexed to 
the report since they do not contain amendments.): 

Description, pages: 

1 -25 as originally filed 

Claims, No.: 

1-28 as originally filed 

Drawings, sheets: 

1/9-9/9 as originally fil<=d 

2. The amendments have resulted in the cancellation of: 

□ the description, pages: 

□ the claims, Nos.: 

□ the drawings, sheets: 

3. □ This report has been established as if (some of) the amendments had not been made, since they have been 

considered to go beyond the disclosure as filed (Rule 70.2(c)): 

4. Additional observations, if necessary: 

III. Non-establishment of opinion with regard to novelty, inventive step and industrial applicability 

The questions whether the claimed invention appears to be novel, to involve an inventive step (to be non-obvious), 
or to be industrially applicable have not been examined in respect of: 

□ the entire international application. 
E3 claims Nos. 1 - 22. 

because: 

Form PCT/lPEA/409 (Boxes l-VIII, Sheet 1) (January 1994) 



INTERNATIONAL PRELIMINARY 
EXAMINATION REPORT 



International application No. PCT/GB99/01 506 



El the said international application, or the said claims Nos. 1 -21 relate to the following subject matter which 
does not require an international preliminary examination {specify): 

see separate sheet 

□ the description, claims or drawings {indicate particular elements be/ow) or said claims Nos. are so unclear 
that no meaningful opinion could be formed {specify): 



□ the claims, or said claims Nos. are so inadequately supported by the description that no meaningful opinion 
could be formed. 

H no international search report has been established for the said claims Nos. 22 . 



V. Reasoned statement under Article 35(2) with regard to novelty, inventive step or industrial 
applicability; citations and explanations supporting such statement 



1. Statement 
Novelty (N) 

Inventive step (IS) 



Yes: Claims 27 



No: 



No: 



Claims 23 - 26 , 28 



Yes: Claims 
No: Claims 27 



Industrial applicability (IA) Yes: Claims 



Claims 1 - 21 



2. Citations and explanations 
see separate sheet 



VII. Certain defects in the international application 

The following defects in the form or contents of the international application have been noted: 
see separate sheet 



Form PCT/IPEA/409 (Boxes l-VIII, Sheet 2) (January 1994) 



INTERNATIONAL PRELIMINARY International application No. PCT/GB99/01 506 

EXAMINATION REPORT - SEPARATE SHEET 



The following documents cited in the International Search report 
will be referred to by means of the following appellation: 
D1 : EP-A-410 045 

D1 is also cited page 1 as US-A-5 097 326 

D2 : WO-A-82-395 
D3 : US-A-5 310 962 

The drawings were filed before the date for demand of examination 8 Dec. 1999 
and are therefore considered as "original" . 



Ill 

Being blind or missing a leg need not be a disease , but a method 
which makes the invalid person walk again is a therapy . 
A leg-prostheses is patentable but the method for mounting it 
in most european countries is not patentable . 

Therefore the method of claims 1 - 21 enabling a blind person to see is 
a therapy even if the method may not give the blind his full eyesight back 
("person" in the first line of claim 1 encompasses blinds , see page 1 line 5 : 
"enable blind persons to see" or page 1 midst : "enable a blind subject ..." 
or page 6 second line : "... recognition of complex shapes"). 
Therefore claims 1-21 are not examined due to Rule 67.1. iv PCT 

If the sole purpose of the method were to make an image recognition-test with 
normal-seeing persons being blind-folded , then the method would fall under 
Rule 67.1. iii PCT (methods of ... playing games). 

Surely no technical problem is solved (Rule 5.1.a.iii PCT) by playing music for 
a normal-seeing person to make him recognize a picture which he might as 
well see with his own eyes. 

Instead of a method , an encoder may be defined by the manner it encodes. 



Form PCT/Separate Shoet/409 (Sheet 1) (EPO-April 1997) 



INTERNATIONAL PRELIMINARY International application No. PCT/GB99/01 506 
EXAMINATION REPORT - SEPARATE SHEET . 



V 

Independent claim 23 is unclear for two reasons , Article 6 PCT : 
Only two types of patent claims exist : 

- claims for a physical entity (product , apparatus) 

- claims for an activity (method , use) 
see PCT-Guideiines C-lll 3.1 

A claim for a physical entity (device) should not contain features of a method. 

Therefore the reference in independent claim 23 (for a device) to features 

from method-claims 1-22 is not permissible. 

Therefore in claim 23 fourth line: 

"... according to any of claims 1 to 22" 

is interpreted as : 

"... contained within an image" , see claim 1. 

In claim 23 last line "the musical sequence" has no reference in the claim. 

Probably is meant the musical sequencies in claim 1 , but this again is 

a mixture of device-features and method-features. 

Therefore the last line of claim 23 is interpreted as: 

"... playing means (84) for playing musical sequencies corresponding 

an encoding of the features of the image" . 



L 



Form PCT/Separate SheeV409 (Sheet 2) (EPO- April 1997) 



INTERNATIONAL PRELIMINARY International application No. PCT/GB99/01 506 

EXAMINATION REPORT - SEPARATE SHEET 



V .2 

In the following claim 23 is interpreted with the above two corrections. 
D1 discloses all features of claim 1 : 

- a device for enabling a person to visualize images 

(D1 page 1 first lines : "... converting visual images into acoustical 
representations ... for totally blind ... persons") 

- imaging means 

(24 in D1 fig 2 is a camera see P.3 L.53) 

- encoding means 

(26 is an image-processing unit P.3 L.54 which converts 
analog image-signals from the camera and sensor 22 , obviously 
relating to features of an image, into digital signals , ie. 
the analog-signals are encoded) 

- playing means 

(the data-processor 30 generates a digitized waveform which is 
D-A-converted 32 and outputted as a musical sequence 16 fig 1 
see last lines on page 3) 

Thus the present application does not satisfy the criterion set forth in 
Article 33(2) PCT because the subject-matter of claim 1 is not new 
Rule 64 PCT. 

V .3 

The camera 24 in Dl can be a video camera D1 P.9 L.12 or 

a CCD-camera P.9 LI 3 

Therefore claims 24 and 25 are not new. 



Form PCT/Separate SheeV409 {Sheet 3) (EPO- April 1997) 



INTERNATIONAL PRELIMINARY International application No. PCT/GB99/01 506 
EXAMINATION REPORT - SEPARATE SHEET 



V .4 

Image processor 26 in D1 fig 2 is shown in details in fig 4 

P.7 L.30 , which is a microprocessor, not a mainframe see P.4 L.15 : 

"... a fast mainframe would have been needed" 

Therefore claim 26 is not new 

V .5 

Head-phones 54 in D1 fig 7 obviously can be replaced by an ear-piece 
Therefore claim 27 is not inventive 

f 

V .6 

The device in D1 is hand-held P.4 L.16 "portable" 
Therefore claim 28 is not new 

V .7 

Reference is also made to D2 figure 1 and to D3 figure 5 
which also show all features of claim 1 of the invention. 

VII 

The features of the claims are not provided with reference signs 
placed in parentheses (Rule 6.2(b) PCT). 



Form PCT/Separate Sheet/409 (Sheet 4) (EPO- April 1997) 



INTERNATIONAL PRELIMINARY International application No. PCT/GB99/01 506 

EXAMINATION REPORT - SEPARATE SHEET 



VII .2 

AN essential features should be in any independent claim see 
PCT-Guidelines C-lll 4.4 

On page 1 midst it is said to be an advantage of the invention that it 
incorporates a system for feature-extraction , but there is no 
extraction-machinery in independent claim 23. 

VII .3 

Contrary to the requirements of Rule 5.1(a)(ii) PCT, the relevant background art 
disclosed in the documents DT D2 and D3 

is not mentioned in the description, nor are these documents identified therein. 

# 



Form PCT/Separate Sheet/409 (Sheet 5) (EPO-April 1997) 



NT COOPERATION TREATY 



From the 

INTERNATIONAL PRELIMINARY EXAMINING AUTHORITY 



To: 



MCNEIGHT, David Leslie 
McNEIGHT & LAWRENCE 
Regent House 
Heaton Lane 

Stockport, Cheshire SK4 IBS 
GRANDE BRETAGNE 



PCT 



NOTIFICATION OF RECEIPT 
OF DEMAND BY COMPETENT INTERNATIONAL 
PRELIMINARY EXAMINING AUTHORITY 

(PCT Rules 59.3(e) and 61.1(b), first sentence 
and Administrative Instructions, Section 601(a)) 



Date of mailing 
( day f month /year) 



28.12.99 



Applicant's or agent's file reference 

M98/0209/PCT 



IMPORTANT NOTIFICATION 



International application No. 
PCT/GB 99/ 01506 



International filing date (dayjmonthjyear) 
12/05/1999 



Priority date ( day /month /'year) . 
12/05/1998 



Applicant 

UNIVERSITY OF MANCHESTER INST. . . . TECHNOLOGYetal . 



The applicant is hereby notified that this International Preliminary Examining Authority considers the following date as the 
date of receipt of the demand for international preliminary examination of the international application: 

08/12/1999 



This date of receipt is: 

p^the actual date of receipt of the demand by this Authority (Rule 61.1(b)). 

| | the actual date of receipt of the demand on behalf of this Authority (Rule 59.3(e)). 



□ 



| | the date on which this Authority has, in response to the invitation to correct defects in the demand 
(Form PCT/I PEA/404), received the required corrections. 

ATTENTION: That date of receipt is AFTER the expiration of 19 months from the priority date. Consequently, the 
election(s) made in the demand does (do) not have the effect of postponing the entry into the national phase until 30 
months from the priority date (or later in some Offices) (Article 39(1)). Therefore, the acts for entry into the national 
phase must be performed within 20 months from the priority date (or later in some Offices) (Article 22). For details, see 
the PC T Applicants Guide, V olume 1 1 . 



| | ( If applicable) This notification confirms the information given by telephone, facsimile transmission or in person 



4. Only where paragraph 3 applies, a copy of this notification has been sent to the International Bureau. 




Name and mailing address of the I PEA/ 



European Patent Office 
Alll D -80298 Munich 

3))§ Tel. ( + 49-89) 2399-0, Tx: 523656 epmu d 
Fax: ( + 49-89) 2399-4465 



Form PCT/IPEA/402 (July 1998) P20452 



(22/12/1999) 



Authorized officer 
SCHWARZKOPF G B 

Tel ( + 49-89) 2399-8SS3 



Submission to Enter National 
Phase Under 35 US 371 



f4^ent cooperation trea© 



PCT/GB99/01 



From the INTERNATIONAL BUREAU 



PCT 

NOTIFICATION CONCERNING 
SUBMISSION OR TRANSMITTAL 
OF PRIORITY DOCUMENT 

(PCT Administrative Instructions, Section 411) 


To: 

MCNEIGHT, David, Leslie 
McNeight & Lawrence 
Regent House 
Heaton Lane 
Stockport 

Phochiro 1 DC 

ROYAUME-UNI 


Date of mailing (day/month/year) 
05 July 1999 (05.07.99) 




Applicant's or agent's file reference 
M98/0209/PCT 


IMPORTANT NOTIFICATION 


international application No. 
PCT/GB99/01506 


International filing date (day/month/year) 
12 May 1999 (12.05.99) 


International publication date (day/month/year) 
Not yet published 


Priority date (day/month/year) 

12 May 1998(12.05.98) 


Applicant 

UNIVERSITY OF MANCHESTER INSTITUTE OF SCIENCE AND TECHNOLOGY et al 



1. The applicant is hereby notified of the date of receipt (except where the letters "IMR" appear in the right-hand column) by the 
International Bureau of the priority document(s) relating to the earlier application(s) indicated below. Unless otherwise 
indicated by an asterisk appearing next to a date of receipt or by the letters "NR", in the right-hand column, the priority 
document concerned was submitted or transmitted to the International Bureau in compliance with Rule 17.1(a) or (b). 

2. This updates and replaces any previously issued notification concerning submission or transmittal of priority documents. 

3. An asterisk*) appearing next to a date of receipt, in the right-hand column, denotes a priority document submitted 
or transmitted to the International Bureau but not in compliance with Rule 17.1(a) or (b). In such a case, the attention 
of the applicant is directed to Rule 17.1(c) which provides that no designated Office may disregard the priority claim 
concerned before giving the applicant an opportunity, upon entry into the national phase, to furnish the priority document 
within a time limit which is reasonable under the circumstances. 

4. The letters "NR" appearing in the right-hand column denote a priority document which was not received by the International 
Bureau or which the applicant did not request the receiving Office to prepare and transmit to the International Bureau, 

as provided by Rule 17.1(a) or (b), respectively. In such a case, the attention of the applicant is directed to Rule 17.1(c) which 
provides that no designated Office may disregard the priority claim concerned before giving the applicant an opportunity, 
upon entry into the national phase, to furnish the priority document within a time limit which is reasonable under the 
circumstances. 

Priority date Priority application No. Country or regional Office Date of receipt 

nr PCT receiving Office of priority d ocument 

12 May 1998(12.05.98) 9809986.4 GB 25 June 1999 (25.06.99) 





Authorized officer l 


The International Bureau of WIPO 


34, chemin des Colombettes 


Carios Naranjo 


1211 Geneva 20, Switzerland 


Facsimile No. (41-22) 740.14.35 


Telephone No. (41 -22) 338.83.38 



Form PCT/IB/304 (July 1998) nn7717fiZ1 

Submission to Enter National 
Phase Under 35 US 371 



P^ENT COOPERATION TREA^ 



PCT/GB99/01 



From the INTERNATIONAL BUREAU 



PCT 



INFORMATION CONCERNING ELECTED 
OFFICES NOTIFIED OF THEIR ELECTION 



(PCT Rule 61.3) 



Date of mailing (day/month/year) 
14 January 2000 (14.01 .00) 



To: 



MCNEIGHT, David, Leslie 

McNeight & Lawrence 

Regent House 

Heaton Lane 

Stockport 

Cheshire SK4 1BS 

ROYAUME-UNI 



Applicant's or agent's file reference 
M98/0209/PCT 


IMPORTANT INFORMATION 


International application No. 
PCT/GB99/01506 


International filing date (day/month/year) 
12 May 1999(12.05.99) 


Priority date (day/month/year) 
12 May 1998(12.05.98) 


Applicant 

UNIVERSITY OF MANCHESTER INSTITUTE OF SCIENCE AND TECHNOLOGY et al 



1. The applicant is hereby informed that the International Bureau has, according to Article 31 (7), notified each of the following 
Offices of its election: 

AP :GH,GM,KE,LS,MW,SD,SL,SZ,UG,ZW 

EP lAT^E^CH^DE^K^S^F^FR.GB^RJEJT^aMCNL^T^SE 

National lAU^BG^BR^CN^C^DEJLJ^KP^KR.MN.NO.NZ.PURO.RU.SE^K^US 

2. The following Offices have waived the requirement for the notification of their election; the notification will be sent to them 
by the International Bureau only upon their request: 

EA :AM,AZ3Y,KG,KZ,MD,RU/TJ,TM 

OA : BF,B J,CF,CG,CI,CM,GA,GN,GW,ML,MR,NE,SN,TD,TG 

National lAE^AM^AZ^BA.BB^Y^H^U^K^^ES^FI^GB^GD^GE^H^GM^HR^U, 

IDJN^S^E^G^^LC^KXR^S^T.LU^^MD^G^^MW^XPT^SD^SG^^SLJJ, 

TM/TR,TT,UA,UG,UZ,VN,YU,ZA,ZW 

3 The applicant is reminded that he must enter the "national phase" before the expiration of 30 months from the Priority date 
before each of the Offices listed above. This must be done by paying the national fee(s) and furnishing , if prescribed, a 
translation of the international application (Article 39(1 )(a)), as well as, where applicable, by furnishing a translation of any 
annexes of the international preliminary examination report (Article 36(3)(b) and Rule 74.1). 

Some offices have fixed time limits expiring later than the above-mentioned time limit. For detailed informati on about the 
applicable time limits and the acts to be performed upon entry into the national phase before a particular Office, see Volume I 
of the PCT Applicant's Guide. 

The entry into the European regional phase is postponed until 31 months from the priority date for all States designated for 
the purposes of obtaining a European patent. 



The International Bureau of W1PO 
34, chemin des Colombettes 
121 1 Geneva 20, Switzerland 



Facsimile No. (41-22) 740.14.35 



Authorized officer: 

Olivia RANAIVOJAONA 

Telephone No. (41-22) 338.83.38 
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Form PCT/IB/332 (September 1997) 



3053019 

Submission to Enter National 
Phase Under 35 US 371 



^ATEMT COOPERATION TFtEA^ 



From the - 
INTERNATIONAL PRELIMINARY EXAMINING AUTHORITY 



To: 

MCNEIGHT, David Leslie 
McNeight & Lawrence 
Regent House 
Heaton Lane 
Stockport 

Cheshire SK4 1BS 
GRANDE BRETAGNE 



PCT 



NOTIFICATION OF TRANSMITTAL OF 
THE INTERNATIONAL PRELIMINARY 
EXAMINATION REPORT 

(PCT Rule 71.1) 



Date of mailing 
(day/month/year) 



08.09.2000 



Applicant's or agenf s file reference 
M98/0209/PCT 


IMPORTANT NOTIFICATION 


International application No. 
PCT/GB99/01506 


Internationa! filing date (day/month/year) 
12/05/1999 


Priority date (day/month/year) 
12/05/1998 


Applicant 

UNIVERSITY OF MANCHESTER INST.:. TECHNOLOGYetal. 



1 . The applicant is hereby notified that this International Preliminary Examining Authority transmits herewith the 
international preliminary examination report and its annexes, if any, established on the international application. 

2. A copy of the report and its annexes, if any, is being transmitted to the International Bureau for communication 
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(Earliest) Priority Date (day/month/year) 

12/05/1998 


Applicant 

UNIVERSITY OF MANCHESTER INST TECHNO LOGYeta 1 . 



This International Search Report has been prepared by this International Searching Authority and is transmitted to the applicant 
according to Article 1 8. A copy is being transmitted to the International Bureau. 



_ sheets. 



This International Search Report consists of a total of 4 

| X | It is also accompanied by a copy of each prior art document cited in this report. 



Basis of the report 

a. With regard to the language, the international search was carried out on the basis of the international application in the 



language in which it was filed, unless otherwise indicated under this item. 

□ 



the international search was carried out on the basis of a translation of the international application furnished to this 
Authority (Rule 23.1 (b)). 

With regard to any nucleotide and/or amino acid sequence disclosed in the international application, the international search 

was carried out on the basis of the sequence listing : 

| | contained in the international application in written form. 

filed together with the international application in computer readable form. 
| | furnished subsequently to this Authority in written form. 
[ | furnished subsequently to this Authority in computer readble form. 

| [ the statement that the subsequently furnished written sequence listing does not go beyond the disclosure in the 

international application as filed has been furnished. 
| [ the statement that the information recorded in computer readable form is identical to the written sequence listing has been 

furnished 

[X| Certain claims were found unsearchable (See Box I). 
| | Unity of Invention is lacking (see Box II). 



4. With regard to the title, 

[X] the text is approved as submitted by the applicant. 

| | the text has been established by this Authority to read as follows: 



With regard to the abstract, 

|"X"| the text is approved as submitted by the applicant 

□ the text has been established, according to Rule 38.2(b), by this Authority as it appears in Box III. The applicant may, 
within one month from the date of mailing of this international search report, submit comments to this Authority 

The figure of the drawings to be published with the abstract is Figure No 



| | as suggested by the applicant. Q None of the figures. 

\%\ because the applicant failed to suggest a figure. 
| | because this figure better characterizes the invention. 
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Box I Observations where certain claims were found unsearchable (Continuation of item 1 of first sheet) 



This International Search Report has not been established in respect of certain claims under Article 1 7(2)(a) for the following reasons: 
1. Claims Nos.: 

because they relate to subject matter not required to be searched by this Authority, namely: 



2. X Claims Nos.: 22 

because they relate to parts of the International Application that do not comply with the presented requirements to such 

an extent that no meaningful International Search can be carried out, specifically: 

The matter claimed in claim 22 is not clear. See articles 5 and 6. 



3. Claims Nos.: 

because they are dependent claims and are not drafted in accordance with the second and third sentences of Rule 6.4(a). 



Box il Observations where unity of invention is lacking (Continuation of item 2 of first sheet) 



This International Searching Authority found multiple inventions in this international application, as follows: 



1 , I I As all required additional search fees were timely paid by the applicant, this International Search Report covers all 
' ' searchable claims. 

2. [^] As all searchable claims could be searched without effort justifying an additional fee, this Authority did not invite payment 

of any additional fee. 



3. I I As only some of the required additional search fees were timely paid by the applicant, this International Search Report 
I ' covers only those claims for which fees were paid, specifically claims Nos.: 



4. I I No required additional search fees were timely paid by the applicant. Consequently, this International Search Report is 
' restricted to the invention first mentioned in the claims; it is covered by claims Nos.: 



Remark on Protest | | The additional search fees were accompanied by the applicant's protest. 

j j No protest accompanied the payment of additional \ 



I search fees. 



Form PCT/1SA/21 0 (continuation of first sheet (1 )) (July 1 998) 



